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Introduction 

Endoscopic  approaches  and  minimally  invasive  techniques  are  now  permitting  visualization  of 
previously  inaccessible  regions  of  the  body  using  endoscopes  which  are  fract’^ns  of  millimeters  in 
size.  But  there  remain  vital  areas  of  the  boy,  within  solid  structures  like  the  temporal  bone,  which 
can't  be  visualhusd  at  current  levels  of  endoscopic  sophistication. 

Using  the  next  generation  of  imaging  and  dintal  fusion  capability,  radiologists  are  developing  tools 
to  br^  through  diese  endoscopic  barriers.  Some  of  the  most  promising  recent  accomplishments 
are  in  the  area  of  three  dimensional  reconstruction,  fusion  and  navigation  throi  gh  CT  and  MR  data. 
Investigations  in  progivss  include  those  in  neurosurgery  (1),  gastrointestinp’  sur^ry  (2)(3)  and 
endoscopic  sinus  surgery  (4).  A  commercial  product  for  ar^oscopic  procedures  is  even  available 
(S).  These  ciq)abilities  are  b^g  used  in  pre-operative  planning  for  unusual  and  difficult 
presentations,  the  development  of  iimovative  procedures  as  well  as  for  the  integration  of  new 
applications  with  advanced  technologies. 

Based  upon  these  successes  and  sophisticated  flight  tracking  programs  from  the  military,  virtual 
anatomical  environments  created  with  specific  patient  data  can  now  be  “flown"  through  and 
features  “targeted"  with  nearly  the  realism  available  to  modem  fighter  pilots. 

There  is  the  added  advantage  that  with  the  virtual  anatomy,  point-of-view  is  not  limited  by  physical 
boundaries.  For  example,  tone  and  delicate  membranes  can  be  flown  through  non-invasively  and 
perspectives  achieved  that  are  impossible  even  in  open  surgery.  This  approach  was  discover^ 
separately  and  simultaneously  by  Lxirensen  (6)  Geiger  and  Kikinis  (7)  and  Vining  (8).  The 
following  is  the  first  report  of  this  application  to  the  fields  of  otologic  and  neuro-otologic  surgery. 

Body 

Materials  and  Methods 

The  four  main  stages  involved  in  the  creation  of  a  virtual  otoscopic  flythrou^  are  slice  acquisition, 
segmentation,  generation  of  three-dimensional  models,  and  flight  path  planning. 

During  slice  acquisition,  cross-sectional  images  are  acquired  using  a  CT  or  MR  scanner  and 
transferrto  via  LAN  to  woilrstations  in  the  lab.  For  this  study,  a  formalin  fixed  specimen  was 
positioned  in  a  CT  scanner  (Siemens  Somatom+,  Erlangen,  Germany)  and  58  in^vidual 
cross-sections  were  acquired  with  a  field  of  view  of  71  millimeters.  Slice  thickness  and  table  feed 
fimm  slice  to  slice  were  1  millimeter.  To  miniirJze  the  appearance  of  slice  artifacts  in  tlie  final 
reconstruction,  the  slices  were  post-processed  using  a  linear  weighted  maximum-likelihood 
interpolation  technique  to  generate  a  new  inteipolated  slice  series  of  180  slices  with  a  smaller  slice 
spacing  of  0.?2  millimeters. 

In  the  segmentation  process,  structures  that  we  want  to  appev  in  the  three  dinwnsiond 
reconstruction  are  itontified  and  labeled,  transforming  the  original  ^y-scale  images  into 
"labelmaps"  so  that  the  pixels  belonging  to  each  structure  are  identified  by  a  unique  number.  In 
our  lab,  this  process  is  performed  using  MrX  (General  Electric  Medical  Systems,  Milwaukee,  WI)> 
a  research  program  originally  developto  by  General  Electric's  Corporate  Research  and 
Devdopment  Onter  in  Schenectady  and  sipificantly  expanded  in  our  lab.  MrX  integrates  a  wide 
variety  of  image  processing  tools  ato  algorithms  to  provide  a  computer-assisted  segmentation 
environment  For  larger  structures  that  exhibit  good  contrast  in  the  slice  images,  the  segmentation 
process  is  almost  completely  automated;  for  the  finer  details,  interactive  editing  and  anatomical 
expertise  are  required. 
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For  the  3-D  visualization,  we  generated  surface  models,  which  allow  visualization  of  the  outer 
surface  of  solid  objects,  because  the  rendering  of  these  models  is  efficient  enough  to  permit 
animation-speed  display  using  desktop  hardware.  The  surface  model  is  represented  by  a  mesh  of 
triangles  that  accurately  approximate  the  shape  of  the  object  being  inodeled  An  algorithm  known 
as  "Marching  Cubes"  (1 1)(12)  produces  the  mesh  for  each  of  the  structures  labeled  in  the 
segmented  slices.  The  resulting  nnxlels  often  contain  several  hundred  thousand  triangles,  so  they 
are  processed  by  a  second  algorithm,  '^Triangle  Decimation,*'  (9)  which  replaces  adjacent  triangles 
with  single  latter  triangles  where  this  is  possible  without  altering  the  geometry  of  the  model 
beyond  a  speckled  tolerance.  This  step  is  required  to  reduce  the  size  aj^  complexity  of  the  fmal 
representation  to  a  point  where  rendering  sp^  is  acceptable  for  animation. 

We  used  a  Sun  SparcStation  20  (Sun  Microsystems,  Mountain  View,  CA)  equipped  with  a  ZX 
graphics  accelerator  (13)  as  the  underlying  hardware  for  the  rendering  and  animation  tasks.  This 
configuration  rendenxl  our  data  (consisting  of  148136  triangles)  at  an  average  speed  of  1.8  frames 
per  second  during  the  flythrough.  For  stereo  viewing,  we  utiliz^  the  stereo  capabilities  of  the  ZX 
board  in  conjunction  with  stereo  glasses  and  a  stereo  controller  (Qystal  Eyes,  Stereographies 
Corp.,  CA). 

The  generation  and  interactive  display  of  the  virtual  otoscopy  utilize  programs  written  in  LYMB,  a 
high-level  object-oriented  language  developed  at  GE.  (10)  LYMB  incorporates  a  broad  array  of 
functionality  with  an  emphasis  on  elements  that  facilitate  scientific  visualization,  modeling  and 
animation,  particularly  within  the  medical  domain.  The  LYMB  rendering  module  uses  a  "virtual 
camera"  abstraction  to  facilitate  intuitive  specification  of  different  views  of  the  models.  Using  a 
"camera  controller"  interface,  the  user  is  afforded  complete  control  of  the  position  and  focal  point 
of  the  virtual  camera  within  the  model. 

We  employed  a  keyframing  technique  to  generate  the  animation.  In  this  process,  the  animator  uses 
the  camera  controller  to  specify  the  views  that  should  appear  in  the  animation,  in  sequence.  The 
computer  records  these  views  and  provides  smooth  transitions  between  them  by  generating  the 
intermediate  frames.  The  resulting  data  file  is  a  LYMB-parsable  script,  so  the  user  can  easily  edit 
the  animation  and  fine-tune  it  using  the  full  versatility  of  the  LYMB  language.  At  this  stage,  we 
added  text  labels  and  ruadc  certain  structures  transparent  for  portions  of  the  virtual  otoscopy. 

Results 

We  generated  a  648  frame  flythrough  of  the  middle  and  inner  ear  using  the  procedures  described  in 
the  previous  section.  The  animation  begins  by  approximating  the  view  obtained  during  a 
conventional  otoscopy,  approaching  the  extemai  auditory  canal.  However,  in  contrast  to  the  real 
procedure,  we  choose  to  remove  the  skin  (see  figures  1  and  2).  As  a  next  step,  however,  the 
virtual  camera  passes  tluough  the  eardrum  and  changes  to  90  degree  optics  for  an  endoscopic 
view. 

The  space  and  maneuverability  constraints  of  a  conventional  procedure  are  absent,  of  course,  as  is 
apparent  when  the  virtual  camera  backs  into  a  recess  of  the  middle  ear  cavity  for  a  clear  view  of  the 
incus,  malleus,  stapes  and  stapedius  muscle.  The  camera  then  flies  along  the  middle  ear  cavity  for  a 
short  distance  until  the  cavity  walls  dissolve  to  reveal  the  inner  ear  structures.  The  view  now 
includes  the  facial  nerve,  the  geniculate  ganglion,  the  greater  petrosal  foramen,  the  carotid  artery, 
the  internal  auditory  and  the  cochlea  and  semi-circular  canals,  all  identiEed  by  different  colors  (see 
figures  3). 


Now,  outside  the  restrictive  confines  of  the  middle  ear  cavity,  we  reven  to  ordin^  30  degree 
optics  to  eliminate  the  distoition  along  the  peripheiy  caused  by  the  wider-angle  view  (see  nguie 
4'6).  At  this  point  text  labels  are  inserted  that  identify  the  structures,  and  the  virtual  camera  moves 
around  and  among  the  structures  for  a  variety  of  close-up  views.  Finally,  the  camera  bacl»  away 
for  a  view  from  inside  the  skull  where  the  entry-points  of  the  caiotid  artery  and  facial  nerve  are 
visible.  During  the  anim^ition  there  is  access  to  the  original  CTT  data.  A  marker  can  be  set  in  either 
the  CT  data  or  the  3d  renderings  and  will  be  visible  on  the  other  views. 

This  animation  is  fully  interactive  when  displayed  in  the  LYMB  environment;  the  user  can  stop  the 
animation  at  anv  point  and  use  a  variety  of  controls  to  manipulate  the  virtual  camera  or  alter  any  of 
the  attributes  of  a  particular  structure.  The  animation  can  then  be  resumed  or  restarted.  Single  frame 
stepping  is  also  available.  Finally,  stereo  viewing  is  possible  using  the  stereo  mode  on  the  ZX 
accelerator,  affording  the  viewer  an  even  more  realistic  representation  of  the  spatial  configuration 
of  the  models. 

Discussion 

The  virtual  otoscopy  demonstrates  the  application  of  interaedve  flythrough  visualization  techniques 
to  a  particularly  difficult  area  of  the  anatomy,  and  reveals  a  trenr.endous  potential  in  both  clinical 
and  educational  settings. 

This  type  of  visualization  has  obvious  applications  in  surgical  planning,  both  pre-operatively  and 
intra-operadvely.  While  the  virtual  otoscopy  is  still  in  the  research  phase,  our  lab  regularly  uses 
3-D  reconstructions  both  for  surgical  planning  and  interactively  during  surgery,  predominantly  in 
the  neurosurgical  domain. 

As  an  educational  tool,  flythrough  visualizations  provide  drantatic  improvements  over  textbook 
illustrations  and  photos,  particularly  when  applied  to  complex  anatomy  like  that  of  the  middle  and 
inner  ear.  The  flythrough  provides  3-D  cues  that  are  not  otherwise  present  so  that  the  location  and 
anatomical  relationships  of  structures  in  a  region  as  complex  as  the  ear  are  more  easily  grasped. 

The  addition  of  text  labels  further  enhances  ^e  educational  value  of  this  type  of  display. 

Once  the  models  and  animations  are  produced,  using  LYMB,  it  is  possible  to  generate  an  MPEG 
"movie"  fix)m  the  flythrough  which  is  not  interactive,  but  captures  the  essential  characteristics  of 
flythrough  visualization.  Tlus  alternative  format  can  be  displayed  on  virtually  any  personal 
computer,  which  could  make  widespread  educational  application  both  practical  and  economical. 
The  MPEG  format  permits  surgeons  to  further  study  cases  on  their  own  personal  computer, 
without  having  to  utilize  special  equipment  which  may  be  less  convenient  or  accessible. 

Further  improvements  in  virtual  reality  techniques  and  stereoscopic  visualization  tools  will  likely 
enhance  this  new  image  representation  method.  A  split  screen  could  display  the  endoscopic  view 
simultaneously  with  a  view  from  an  outside  perspective  and  a  display  of  the  nearest  cross-sectional 
slice.  Two-dimensional  and  3-D  maricers  could  track  the  position  of  the  virtual  endoscope,  and 
provide  a  cross-referencing  capability  between  the  views.  In  the  future,  advanced  virtual  reality 
interface  devices  may  furtiwr  enhance  the  process  of  exploration  and  interactive  navigation  widiin 
the  models. 


Conclusion 


Virtual  otoscopv  is  clearly  in  its  fomutive  sta^  but  portends  a  potential  beyond  our  current  grasp. 
Segmentation  pr  jcesses  and  pathological  distinctions  are  not  yet  sophisticated  enough  to  generate 
hi^  resolution  images  comparable  to  optical  endoscopy  and  which  enable  the  sometimes  subtle 
distinctions  indicative  of  disease.  Therapeutic  procedures  cannot  be  performed  but  "routine 
screening"  may  soon  be  cost-consciousiy  possible.  In  this  sense,  the  new  capabilities  will  be  more 
an  adjunct  than  a  replacement  For  examination  of  the  middle  ear  and  structures  within  the  temporal 
bone,  an  entirely  new  dimension  of  care  will  be  provided.  The  challenge  is  perhws  not  so  much 
Aat  of  virtual  otoscopy  replacing  standard  procedures  but  rather  who  will  be  performing  it 
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Figure  1; 

Lateral  view  of  the  temporal  bone.  The  external  auditory  canal,  the  mastoid  and  the  tcmporo 
andibular  joint  are  visible.  The  eardrum  is  visible  at  the  bottom  of  the  external  auditory  canal. 
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The  virtual  otoscopic  camera  is  now  inside  the  external  auditory  canal.  The  handle  of  the  malleus  is 
visible  through  the  orange  eardrum. 
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Figure  3-6: 

View  of  the  soft  tissues  of  the  petrosal  bone:  The  facial  nerve  is  colorized  yellow,  the  cochlea  and 
the  semicircular  canals  are  colorized  blue.  The  geniculate  ganglion  is  represented  in  a  tan  color  and 
the  greater  petrosal  foramen  is  colorized  in  a  darker  brownish  hue.  The  carotid  is  colorized  in  red 
and  internal  auditory  canal  and  its  branches  is  colorized  in  an  orange  color.  Of  the  bones  of  the 
middle  car,  incus  and  stapes  are  colored  green,  the  malleus  is  colored  in  a  flesh  tone. 

Figure  3:  A  close  up  view  of  the  geniculate  ganglion.  A  white  spheric  marker  is  positioned  in  the 
ganglion. 
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Figure  4:  A  view  from  farther  away.  The  virtual  camera  that  was  used  to  take  figure  3  is 
represented  as  a  yellow  box  with  a  blue  pyramid  marking  the  upside. 
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Figure  5:  On  this  view  the  geniculate  ganglion  and  the  greater  petrosal  foranien  have  been 
removed,  partially  exposing  the  cochcla. 


V 


•'J 


Rgi]re6 


Figure  6:  The  internal  auditory  has  also  been  removed  to  further  expose  the  cochlea. 
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Figure  7:  Corre^nding  CT  slice.  This  is  the  data  from  wiiich  the  3d  models  were  generated.  The 
bl^k  arrow  points  at  the  geniculate  ganglion. 
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